Papers with reference-free metric
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation (2020.acl-main)
Copied to clipboard
| Challenge: | Standard language generation metrics have been shown to be ineffective for dialog evaluation. |
| Approach: | They propose an unsupervised evaluation metric for dialog that trains unsupervised models to measure several desirable qualities of dialog. |
| Outcome: | The proposed evaluation metric strongly correlates with human judgment on Topical-Chat and PersonaChat. |
HaRiM+: Evaluating Summary Quality with Hallucination Risk (2022.aacl-main)
Copied to clipboard
| Challenge: | Existing summarization models are limited in measuring the factual inconsistency of generated summaries. |
| Approach: | They propose a decoder overconfidence-regularizing objective as a hallucination risk measurement to better estimate the quality of generated summaries. |
| Outcome: | The proposed metric is reference-free and requires no training or modules . it records state-of-the-art correlation to human judgment on three sets of summary-quality annotations. |
Assessing Reference-Free Peer Evaluation for Machine Translation (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods to evaluate machine translation output are based on comparing MT output to one or more reference translations. |
| Approach: | They propose to use probabilities given by a large, multilingual model as a reference-free metric. |
| Outcome: | The proposed model is robust and likely to offer reasonable performance across a broad spectrum of domains and different system qualities. |
CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smells (2025.naacl-long)
Copied to clipboard
| Challenge: | Current review comment evaluation metrics rely on comparisons with a human-written reference for a given code change (also called a diff). |
| Approach: | They propose to use a reference-free metric to measure review quality like conciseness, comprehensiveness, and relevance to compare the quality of code changes with human-written references. |
| Outcome: | The proposed metric can produce fine-grained scores that have the greatest alignment with human judgment and are more sensitive than reference-based metrics. |
MATCH: Task-Driven Code Evaluation through Contrastive Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | GitHub Copilot generates 46% of the code on GitHub. |
| Approach: | They propose a reference-free metric that uses Contrastive Learning to generate meaningful embeddings for code and natural language task descriptions. |
| Outcome: | This paper compares the performance of a new similarity score with existing metrics. |
Reference-based Metrics Disprove Themselves in Question Generation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing metrics for question generation are based on human-written references . however, the results of the metrics on our study disprove the metrics themselves . |
| Approach: | They propose to use human-written references to evaluate question generation . they propose to combine criteria such as naturalness, answerability, and complexity . |
| Outcome: | The proposed model is based on multi-dimensional criteria such as naturalness, answerability, and complexity, utilizing large language models. |
Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing metrics for summarization are reference-based and correlate poorly with relevance . fluency, faithfulness, coherence and relevance are all measures of human evaluation . |
| Approach: | They propose a reference-free metric that correlates well with human evaluated relevance . n-gram importance weighting is used to weight a summary's importance . |
| Outcome: | The proposed metric can be used along reference-based metrics to improve their robustness in low quality reference settings. |